Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment
نویسندگان
چکیده
Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon's Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.
منابع مشابه
Conducting Usable Privacy & Security Studies with Amazon’s Mechanical Turk
Being able to conduct human subjects experiments in a distributed method across the Internet is frequently desirable to support broad tests of usability. Until recently these experiments were commonly advertised in an ad-hoc fashion, using mailing lists, contest sites, and online bulletin boards. Recently Amazon’s Mechanical Turk, a service where users can complete short tasks and receive autom...
متن کاملCrowdsourcing and Creativity
In studying creativity and techniques for enhancing the creative process, we utilized crowdsourcing through Amazon’s Mechanical Turk. Through our studies we found that in its current form, the Mechanical Turk platform has some limitations in supporting creative tasks. In this position paper, we describe our studies and give recommendations for crowdsourcing platforms that foster creativity.
متن کاملTowards a Guideline for Conducting Economic Experiments on Amazon’s Mechanical Turk
During the last decade Amazon Mechanical Turk has evolved to an established platform for conducting behavioral research. However, designing and conducting economic experiments on online labor markets remains a complex and ambitious task. In comparison to laboratory environments, a set of specific challenges (such as synchronization and control) has to be thoroughly addressed. In order to suppor...
متن کاملReview Paper: The Bed Incline and Prevention of Ventilator-Associated Pneumonia
Background: Ventilator-Associated Pneumonia (VAP) is the most common nosocomial (hospital acquired) infection among patients undergoing mechanical ventilation. It increases mortality rate, duration of mechanical intubated ventilation, and hospitalization in the Intensive Care Units (ICUs). This review study aims to determine the proper gradient of a hospital bed in preventing VAP in patients ho...
متن کاملHow to use Mechanical Turk for Cognitive Science Research
In this half-day tutorial we will describe a new tool that has emerged in the last 5 years for conducting online behavioral research: crowdsourcing platforms. The term crowdsourcing has its origin in an article by Howe (2006), who defined it as a job outsourced to an undefined group of people in the form of an open call. One of the main benefits of these platforms to behavioral researchers is t...
متن کامل